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ABSTRACT 

Three goals must be met in order for the National 
Center for Education Statistics (NCES) to improve the quality and 
utility of its data collection: (1) the choice of what to collect 
must be driven by the questions of interest to decisionmakers and the 
public; (2) procedures must insure validity and reliability of the 
data; and (3) the data must be reported in ways that facilitate use 
by the intended audience. The reliability of the data reported is the 
biggest challenge facing NCES. Careful choices about what to collect 
and clear reporting cannot compensate for inaccuracy. Reporting data 
should include the following points: (1) description of similarities 
and discrepancies from different sources; (2) indication of how 
various sources were used, when applicable; (3) precise descriptions 
of what the numbers represent and interpretations of how their known 
weakness influence the results; (4) knowledgeable interpretation of 
data; (5) methods of reporting which minimize inappropriate 
comparisons; (6) full and accurate citation of data sources; and (7) 
inclusion of a glossary to help readers. NCES should investigate the 
use of minicomputers for different kinds of assessment instruments 
and should begin to design a computerized data collection system 
utilizing telecommunications capability. Comments on "Indicators of 
Education Status and Trends" are listed with page references. 
(JAZ) 
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Introduction 

The National Center for Education Statistics (NCES) seeks to 
redesign Its data collection efforts to Improve the quality and 
utility of the data for decisionmakers and the general public. 
For this to occur, three goals must be met. First, the choice of 
what data to collect must be driven by the questions of Interest 
to decisionmakers and the public. Second, procedures must be In 
place that insure the data are valid and reliable. Third, the 
data must be reported In viays that facilitate use by the Intended 
audiences. 

For NCES, the second goal Is the most critical. Careful 
choices about what data to collect and clear reporting cannot 
compensate for Inaccurate data. Below I first comment on data 
accuracy and then discuss reporting, citing examples from 
Indicators of Education Status and Trends (January 1985). I then 
list specific comments on Indicators of Education Status and 
Trends with page references. 



Data Accuracy 

The biggest challenge facing NCES Is that of Insuring the 
validity and reliability of the data they report. If the data 
continue to be as Inaccurate In the future as they have In the 
past, all other Issues are moot. The more levels of aggregation 
the data pass through, the more sources for error. Because NCES 
must rely on second, third, and fourth hand data. It Is essential 
tc put Into place a set of procedures designed to check the 
validity and reliability of the data. 

Given the need to rely on data from other sources 
(particularly state administrative data which are notoriously 
Inaccurate), NCES must, at the least, develop a system that 
permits crosschecking the data with other sources for the same 
information. Judging from the description of current data 
sources, it appears that there are multiple sources of data for 
certain types of information (e.g., data on staffing and teacher 
characteristics reported by the states are also collected in 
NCES's Public School Survey). To the extent thai: multiple data 
sources already exist, NCES should make comparisons across data 
sources and report on both the extent to which discrepancies are 
found and plausible explanations for the discrepancies. NCES 
should exploit all opportunities to corroborate data sources over 
which they have little direct control, such as state 
administrative data and Census data. 

Data for which multiple sources do not currently exist 
should be collected through alternative means designed explicitly 
as a crosscheck. The Fast Response Survey System could readily 
be adapted to this end. Of course, this presumes that care is 

3 

ER|c 143 



taken to ask precise questions. Unfortunately, my only 
experience with NCES Fast Response Survey data suggests that 
these data are often Inaccurate as well. On the one occasion In 
which I used these data (1979), my own telephone surveys 
corroborated by field work produced figures quite different from 
those of NCES (e.g., NCES reported 66 schools In California 
participating In the Schoolwlde Projects Provision of what was 
then ESEA Title I; I located 107 such schools). 

Perhaps NCES could also establish sauaples of schools within 
states (If such a sample does not exist) as an extension of the 
Fast Response Survey System. NCES could also build Items Into 
contracted longitudinal studies for purposes of corroborating 
other data sources. In addition, NCES should be aware of other 
national data collection efforts (particularly federally funded 
studies In education, labor, and health and annual surveys such 
as Gallup, Louis, etc.) and develop agreements for sharing data 
with the funding or data collection agency. 



Reporting 

Given the nature of the data to be collected, and the 
reliance on Indirect sources and multiple levels of aggregation, 
there will always be Issues of validity and reliability. 
However, NCES can take steps (a) to maximize validity and 
reliability and (b) to Inform readers of the weaknesses that 
remain. 

In reporting the data. It Is absolutely essential to have 
Indicators of the validity and reliability the data. As a 
user of data, I am always suspicious of any type of survey data 
or compilation across levels of government. However, when I know 
how the question was asked, I can draw my own conclusions about 
the bias of the responses. When I know the sample size and a 
standard deviation or a confidence Interval, I can draw my own 
conclusions about Its credibility and utility. Through 
television and other media, even lay audiences are accustomed to 
confidence Intervals and other Indicators of measurement error. 

Based on a close reading of the Indicators of Education 
Status and Trends , I urge NCES to consider the following 
recommendations for reporting the data In addition to reporting 
how questions were asked and estimates of reliability. 

1) Following the previous recommendations for corroborating 
data, reported data should Include a brief description of 
the similarities and discrepancies from different sources 
and an analysis of what accounts for the differences. To 
the extent that the discrepancies Influence Interpretation 
of the data, the text should alert the reader to the limits 
on Interpretation. 
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2) When multiple sources are cited, as Is currently the case 
for many of the tables and charts In Indicators , Indicate 
how the multiple sources were used. Multiple sources always 
suggests to me that data were merged without regard to 
comparability. When multiple sources are cited, note which 
data came from which source and, If two sources were merged 
for one estimate, describe how and why this was done. 

3) Precise descriptions of what the numbers represent and 
Interpretations of how their known weaknesses Influence the 
results Is crucial. Some Inaccuracy Is Inevitable In such a 
mammoth system of data collection. No one expects 
otherwise. The utility of the data rests on attaining an 
acceptable level of quality and alerting the reader to Its 
remaining weaknesses. Unfortunately, the text In Indicators 
accomplishes neither of these goals. Sloppy table titles 
and category labels are Inexcusable as Is vague and 
ungraiuwtlcal prose. (See examples below.) Readers must 
understand the limits of the data. 

4) Knowledgable Interpretation of data Is useful to readers; 
uninformed or sloppy Interpretation Is dangerous. NCES 
should provide Interpretation but not without a system that 
corroborates the Interpretatlon(s) . One approach Is to 
create panels of outside reviews In different areas (e.g., 
one for student performance, another for human resources) 
who would comment on a draft of the presentation. Agreement 
Is not essential; presenting conflicting Interpretations Is 
also extremely useful to readers. 

5) Collect and report the data In ways that minimize 
Inappropriate comparisons. For example, comparing states by 
comparing trends over time within states decreases the 
problems due to different measures and definitions. NCES 
does this well In several places. In addition, the text 
around tables and charts should draw the readers' attention 
to the appropriate comparisons. 

6) NCES shoald cite data sources more accurately and fully. 
When "NCES estimates" are cited as the data source, my 
eyebrows go up. All sources should Include dates. 

7) Use the glossary to help readers. Dictionary definitions 
of enrollment, attendance and biology, for example, are not 
helpful. The Glossary should reflect the way questions were 
asked and Indicate differences In definitions across states 
or data sources. 



Comments on "Indicators** 

The concept of a report on Indicators of education status 
and trends Is excellent. This kind of annual report has the 
potential to provide an Invaluable picture of our educational 
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system over time. At the least. It provides a backdrop against 
which to Interpret other data and educational Issues at all 
levels — national, state and local. At the most, the data can 
Inform the public and policymakers about expected trends In such 
critical areas as teachers supply and student enrollment. 

Recognizing that the January 1985 Indicators of Education 
Status and Trends Is a first attempt which seeks reactions from 
potential users, I offer the following specific reactions by 
report section and page. 



Outcomes 

p. 3 I find the NAEP data by assessment area within subject 
particularly Illuminating. I realize presenting subscales 
greatly Increases the quantity of the data. This Is an Instance 
In which some clear text around the tables would be useful. The 
text could report conclusions from an Inspection of more detailed 
tables and reference other publications. (A minor point: 
describing shifts In performance "over the p<-ist decade or so" Is 
misleading, especially for science which Includes no data from 
the past eight years.) 

p. 6 I don't know what to conclude from these data. How were 
these topics measured? Old the Items correspond to e particular 
curriculum? Did they emphasize computation or problem solving? 
Are the numbers medians of 18 means? Do the results look 
different If the comparison Is between the United States and the 
five countries with the same proportion enrolled In math? 

p. 8 Because this Is an area In which many states are changing 
their requlrenents, It would be useful to see shifts over time 
and a breakdown by state. Are these only comprehend. ve high 
schools? 

p. 10 This Is an Instance In which how the question was asked 
and to whom Is critical. Can GEDs be reported separately? (It 
Isn't clear whether they are Included on p. 10 or not). 



Resources 

p. 24 Basing pupil/teacher ratios on all Instructional staff 
results In a gross underestimate of class size. Readers will 
draw from this table an Image that severely distorts reality. 
These data must be presented for teachers with regular classroom 
assignments or not at all. There Is no Indication of what the 
sample Is for the class size data; are they based on regular 
teachers only? 
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Why l8 the research literature mentloni»d here and not 
elsewhere? If research Is mentioned, there should be complete 
citations. I don't think It belongs here; an adequate summary Is 
difficult to do In one sentence. 

"In 1971 and subsequent years, the data by level are 
estimated^" The footnote should say how and why the estimates 
were made. Citing "unpublished data and estimates" only Invites 
suspicion. 

p. 26-27 The headings and labels for the table on p. 26 are a 
good Illustration of the need to use precise language. The title 
Is extremely misleading; this Is not a table about the 
distribution of academic ability In the teaching force. 
Throughout the present report, ryplcally the charts on the right 
are far easier to understand than the tables on the left. Tnls 
Is partly because graphs are often easier to grasp, but more due 
to the clarity of the titles and headings. The title and labels 
of the graph are much more precise and hence clearer than those 
on the left. (E.g., Percent Scoring In Highest Fifth Is much 
clearer than Highest Rank with a footnote saying the sacple was 
broken Into five ranks). 

The content of these two pages also raises questions. 
First, It Is unconscionable to refer to these data as measures of 
the "Quality of the Teaching Force." No single test score can 
capture teacher quality. Moreover, given the small percent of 
teachers who take the SAT (and no Indication of this In the 
table) r I conclude that the numbers are extremely misleading. I 
have no problem with the Idea of looking at teacher performance 
on academic measures, but I need to be convinced that these data 
speak to that issue. Perhaps this is an Instance in which the 
National Longitudinal Study is not the best source of data. 
(The samples seem quite small.) 

p. 28 For data about teacher supply and deiaand to be useful, 
they need to be reported separately for elementary and secondary 
school and by subject area. It would also be useful to see these 
figures by region anc\ the same breakdowns for teachers "teaching 
out of subject" — that is, teaching in fields for which they are 
not certified. The table doesn't state that the entries are in 
thousands . 



Context 

p. 37 Do data on teachers' perceptions of problems exist over 
time? The wording of the second bullet suggests that the 
Metropolitan Life/Harris Survey was conducted In other years as 
well. It would be useful to see trends in these data. Were the 
questions posed to the public and to teachers in the same way? 
If not, since the tables invite comparisons, it would be useful 
to know how they differ. 
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p. 38 Certainly school anvlronnent Indicators are of Interest. 
However, the key factors associated with effective schools are 
not amenable to the kind of neasurement that could be aggregated 
across schools and districts and states. Most of us don't even 
know how to measure them on site. The kinds of perceptions to be 
reported In the 1984 follow-up for High School and Beyond nay 
prove useful. I suspect that the greatest utility will lie In 
the Items that have been asked over time. As with trends within 
state, shifts over time are easier to Interpret than absolute 
levels of factors like "environment conducive to student 
achievement." This suggests a critical need for ongoing 
longitudinal studies of this type. 

p. 40 How many students actually receive these various types of 
services? 

p. 42 Where are the data cited In the text on pages 42 and 437 
To estimate school enrollment trends, are census data available 
on numbers of bables/chlldren ages 0-3? 

p. 44 Referring to state required Carnegie units as "State 
Governance" seems odd. 



Additional data that I would find useful Include: 

-median age of teachers by state 
-data on preschool attendance 
-measures of student mobility /turnover 

-Information on number of hours worked (and types of Jobs) 
for high school students by state and minority status 



Looking to the Future 

The proliferation of microcomputers In district offices and 
schools, combined with growing sophistication about and access to 
telecommunications, has far reaching Implications for future data 
collection. Now Is not the time to Implement such a system; 
neither access to the technology nor user sophistication Is 
sufficiently widespread. But now Is the time to begin to desi;n 
a computerized data collection system utilizing the 
telecommunications capability that most districts and schools 
will have within a decade or less. Such a system will require 
considerable planning and testing; waiting until the technology 
Is completely In place will put NCES a decade behind, if General 
Motors can design a system that automatically translates an 
Individual customer's order Into Instructions for what parts to 
manufacture and Into a custom made car, NCES ought to be able to 
gather basic descriptive Information about our schools through 
similar applications of technology. 

The second application of technology that NCES should now be 
Investigating Is the use of microcomputers for different kinds of 
assessment Instruments. The limits of paper and pencil tests are 
well known. Designing new measures that go beyond simple 
multiple choice questions should be underway. 
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